Apparatus for adjusting coolant flow resistance through liquid-cooled electronics racks

ABSTRACT

Flow restrictors are employed in association with multiple heat exchange tube sections of a heat exchange assembly, or in association with multiple coolant supply lines or coolant return lines feeding multiple heat exchange assemblies. Flow restrictors associated with respective heat exchange tube sections (or respective heat exchange assemblies) are disposed at the coolant channel inlet or coolant channel outlet of the tube sections (or of the heat exchange assemblies). These flow restrictors tailor coolant flow resistance through the heat exchange tube sections or through the heat exchange assemblies to control overall heat transfer within the tube sections or across heat exchange assemblies. In one embodiment, the flow restrictors tailor a coolant flow distribution differential across multiple heat exchange tube sections or across multiple heat exchange assemblies.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. patent application Ser. No.12/556,040, entitled “Apparatus And Method For Adjusting Coolant FlowResistance Through Liquid-Cooled Electronics Rack(s),” filed Sep. 9,2009, and published Mar. 10, 2011 as U.S. Patent Publication No.2011/0056675 A1, and which is hereby incorporated herein by reference inits entirety.

BACKGROUND

The present invention relates in general to a method and apparatus foradjusting coolant flow resistance within one or more liquid-cooledelectronics racks or between multiple electronics racks.

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both module and system level. Increased airflow rates are needed toeffectively cool high power modules and to limit the temperature of theair that is exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power supplies, etc.)are packaged in removable drawer configurations stacked within a rack orframe. In other cases, the electronics may be in fixed locations withinthe rack or frame. Typically, the components are cooled by air moving inparallel airflow paths, usually front-to-back, impelled by one or moreair moving devices (e.g., fans or blowers). In some cases it may bepossible to handle increased power dissipation within a single drawer byproviding greater airflow, through the use of a more powerful air movingdevice or by increasing the rotational speed (i.e., RPMs) of an existingair moving device. However, this approach is becoming problematic at therack level in the context of a computer installation (i.e., datacenter).

The sensible heat load carried by the air exiting the rack is stressingthe availability of the room air-conditioning to effectively handle theload. This is especially true for large installations with “serverfarms” or large banks of computer racks close together. In suchinstallations, liquid cooling (e.g., water or refrigerant cooling) is anattractive technology to manage the higher heat fluxes. The liquidabsorbs the heat dissipated by the components/modules in an efficientmanner. Typically, the heat is ultimately transferred from the liquid toan outside environment, whether air or liquid cooled.

BRIEF SUMMARY

In one aspect, a cooling apparatus for a plurality of electronics racksis provided, wherein each electronics rack includes a heat exchangeassembly. The cooling apparatus includes a coolant distribution unit, aplurality of coolant supply lines, a plurality of coolant return lines,and a plurality of flow restrictors. The coolant distribution unitsupplies cooled system coolant to the heat exchange assemblies of theplurality of electronics racks, and each coolant supply line of theplurality of coolant supply lines is coupled in fluid communication withthe coolant distribution unit and the heat exchange assembly of arespective electronics rack for facilitating supply of system coolantfrom the coolant distribution unit to the respective heat exchangeassembly. Each coolant return line is coupled in fluid communicationbetween the heat exchange assembly of a respective electronics rack andthe coolant distribution unit for facilitating return of exhaustedsystem coolant from the heat exchange assembly to the coolantdistribution unit. In operation, system coolant circulates in a closedloop between the coolant distribution unit and the heat exchangeassemblies via, at least in part, the plurality of coolant supply linesand the plurality of coolant return lines. The plurality of flowrestrictors are associated with at least one of the plurality of coolantsupply lines or the plurality of coolant return lines. Each flowrestrictor is associated with a respective coolant line of the pluralityof coolant supply lines or the plurality of coolant return lines fortailoring coolant flow resistance through the respective heat exchangeassembly. The plurality of flow restrictors tailor coolant flowresistance through at least one of the plurality of coolant supply linesor the plurality of coolant return lines to enhance overall heattransfer through the heat exchange assemblies of the plurality ofelectronics racks.

Further, additional features and advantages are realized through thetechniques of the present invention. Other embodiments and aspects ofthe invention are described in detail herein and are considered a partof the claimed invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 depicts one problem addressed by the present invention, showingrecirculation of airflow patterns in one implementation of a raisedfloor layout of an air-cooled data center;

FIG. 3 is a top plan view of one embodiment of an electronics rack withan air-to-liquid heat exchanger mounted to an outlet door thereof, inaccordance with one aspect of the present invention;

FIG. 4 is a top plan view of one embodiment of a data center employingcooling apparatuses comprising outlet door air-to-liquid heatexchangers, in accordance with an aspect of the present invention;

FIG. 5 is a schematic of one embodiment of a coolant distribution unitto be used in the data center of FIG. 4, in accordance with an aspect ofthe present invention;

FIG. 6 is a partial cross-sectional, elevational view of one embodimentof an electronics rack door and cooling apparatus mounted thereto, takenalong line 6-6 in FIG. 7, in accordance with an aspect of the presentinvention;

FIG. 7 is a cross-sectional, top plan view of the door and coolingapparatus of FIG. 6, taken along line 7-7 in FIG. 6, in accordance withan aspect of the present invention;

FIG. 8 is a schematic of one embodiment of a data center comprising acooling apparatus for distributing coolant flow between electronicsracks of the data center, in accordance with an aspect of the presentinvention;

FIG. 9A is a graph of one embodiment of pressure drop through severalheat exchangers at different load versus coolant flow rates through theheat exchangers, illustrating potential maldistribution of coolant flowacross the heat exchangers of multiple electronics racks of a datacenter, which is addressed in accordance with an aspect of the presentinvention;

FIG. 9B is a graph of pressure drop through several heat exchangers atdifferent load versus coolant flow rates through the heat exchangers,wherein multiple flow restrictors are utilized to tailor coolant flowresistance through the heat exchangers of the multiple electronics racksto ensure that the high heat load electronics rack receives a maximumcoolant flow, in accordance with an aspect of the present invention;

FIG. 10 depicts one embodiment of logic for adjusting coolant flowresistance to multiple heat exchange assemblies of multiple electronicsracks to be cooled, in accordance with an aspect of the presentinvention;

FIG. 11 is a partial cross-sectional, elevational view of theelectronics rack door and cooling apparatus of FIG. 6, with a pluralityof flow restrictors shown disposed at the coolant channel inlets andcoolant channel outlets of the multiple heat exchange tube sections, andtaken along line 11-11 in FIG. 12, in accordance with an aspect of thepresent invention;

FIG. 12 is a cross-sectional, top plan view of the door and coolingapparatus of FIG. 11, taken along line 12-12 in FIG. 11, in accordancewith an aspect of the present invention;

FIG. 13 is a partial cross-sectional, elevational view of one embodimentof an electronics rack door and cooling apparatus mounted thereto withmultiple flow restrictors at the coolant channel inlets of the heatexchange tube sections which comprise different-sized orifice diametersthat tailor coolant flow resistance through the multiple heat exchangetube sections, in accordance with an aspect of the present invention;

FIG. 14 is a schematic of one embodiment of a heat exchange assemblycomprising a heat exchange tube section with a first, fixed diameterflow restrictor at the coolant channel inlet of the heat exchange tubesection and a second, fixed diameter flow restrictor at the coolantchannel outlet of the heat exchange tube section for adjusting coolantflow resistance through the tube section, in accordance with an aspectof the present invention;

FIG. 15A is a schematic of one embodiment of a flow restrictor with afixed orifice diameter for tailoring coolant flow resistance, inaccordance with an aspect of the present invention;

FIG. 15B depicts an alternate embodiment of a flow restrictor with afixed orifice diameter for tailoring coolant flow resistance, inaccordance with an aspect of the present invention;

FIG. 16 is a schematic of an alternate embodiment of a heat exchangeassembly comprising a heat exchanger with a passively controlled,adjustable flow restrictor disposed at the coolant channel inlet of theheat exchange tube section for dynamically adjusting coolant flowresistance through the heat exchange tube section based on sensedairflow temperature across the heat exchange tube section, in accordancewith an aspect of the present invention;

FIG. 17 depicts an alternate embodiment of a heat exchange assemblycomprising a heat exchange tube section with an actively controlled,adjustable flow restrictor at the coolant channel inlet thereof, andpressure and temperature sensors at the coolant channel inlet andcoolant channel outlet of the heat exchange tube section for dynamicallyadjusting the orifice opening size of the actively controlled,adjustable flow restrictor based on sensed pressure and temperature ofcoolant within the heat exchange tube section, in accordance with anaspect of the present invention; and

FIG. 18 depicts one embodiment of logic for controlling coolant flowresistance through multiple heat exchange tube sections of a heatexchange assembly associated with an electronics rack to enhance heattransfer within the multiple heat exchange tube sections of the heatexchange assembly, in accordance with an aspect of the presentinvention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronics system, and may be, for example, a standalone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise multipleelectronics subsystems, each having one or more heat generatingcomponents disposed therein requiring cooling. “Electronics subsystem”refers to any sub-housing, blade, book, drawer, node, compartment, etc.,having one or more heat generating electronic components disposedtherein. Each electronics subsystem of an electronics rack may bemovable or fixed relative to the electronics rack, with the rack-mountedelectronics drawers of a multi-drawer rack unit and blades of a bladecenter system being two examples of subsystems of an electronics rack tobe cooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, a computer system or other electronics unitrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesand memory support dies. As a further example, the electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier. As used herein, “primary heat generating component”refers to a primary heat generating electronic component within anelectronics subsystem, while “secondary heat generating component”refers to an electronic component of the electronics subsystemgenerating less heat than the primary heat generating component to becooled. “Primary heat generating die” refers, for example, to a primaryheat generating die or chip within a heat generating electroniccomponent comprising primary and secondary heat generating dies (with aprocessor die being one example). “Secondary heat generating die” refersto a die of a multi-die electronic component generating less heat thanthe primary heat generating die thereof (with memory dies and memorysupport dies being examples of secondary dies to be cooled). As oneexample, a heat generating electronic component could comprise multipleprimary heat generating bare dies and multiple secondary heat generatingdies on a common carrier. Further, unless otherwise specified herein,the term “liquid-cooled cold plate” refers to any conventional thermallyconductive structure having a plurality of channels or passagewaysformed therein for flowing of liquid coolant therethrough. In addition,“metallurgically bonded” refers generally herein to two components beingwelded, brazed or soldered together by any means.

As used herein, “air-to-liquid heat exchange assembly” means any heatexchange mechanism characterized as described herein through whichliquid coolant can circulate; and includes, one or more discreteair-to-liquid heat exchangers coupled either in series or in parallel.An air-to-liquid heat exchanger may comprise, for example, one or morecoolant flow paths, formed of thermally conductive tubing (such ascopper or other tubing) in thermal or mechanical contact with aplurality of air-cooled cooling fins. Size, configuration andconstruction of the air-to-liquid heat exchange assembly and/orair-to-liquid heat exchanger thereof can vary without departing from thescope of the invention disclosed herein. A “liquid-to-liquid heatexchanger” may comprise, for example, two or more coolant flow paths,formed of thermally conductive tubing (such as copper or other tubing)in thermal communication with each other. Size, configuration andconstruction of the liquid-to-liquid heat exchanger can vary withoutdeparting from the scope of the invention disclosed herein. Further,“data center” refers to a computer installation containing one or moreelectronics racks to be cooled. As a specific example, a data center mayinclude one or more rows of rack-mounted computing units, such as serverunits.

One example of facility coolant and system coolant is water. However,the concepts disclosed herein are readily adapted to use with othertypes of coolant on the facility side and/or on the system side. Forexample, one or more of the coolants may comprise a brine, afluorocarbon liquid, a liquid metal, or other similar coolant, orrefrigerant, while still maintaining the advantages and unique featuresof the present invention. As a specific example, the concepts describedhereinbelow with reference to FIGS. 8-18 employ water as facilitycoolant and a refrigerant as system coolant.

Reference is made below to the drawings (which are not drawn to scale tofacilitate understanding of the invention), wherein the same referencenumbers used throughout different figures designate the same or similarcomponents.

As shown in FIG. 1, in a raised floor layout of an air cooled computerinstallation or data center 100 typical in the prior art, multipleelectronics racks 110 are disposed in one or more rows. A computerinstallation such as depicted in FIG. 1 may house several hundred, oreven several thousand microprocessors. In the arrangement of FIG. 1,chilled air enters the computer room via floor vents from a supply airplenum 145 defined between the raised floor 140 and a base or sub-floor165 of the room. Cooled air is taken in through louvered covers at airinlet sides 120 of the electronics racks and expelled through the back(i.e., air outlet sides 130) of the electronics racks. Each electronicsrack 110 may have an air moving device (e.g., fan or blower) to provideforced inlet-to-outlet airflow to cool the electronic components withinthe drawer(s) of the rack. The supply air plenum 145 providesconditioned and cooled air to the air-inlet sides of the electronicsracks via perforated floor tiles 160 disposed in a “cold” air aisle ofthe data center. The conditioned and cooled air is supplied to plenum145 by one or more conditioned air units 150, also disposed within thecomputer installation 100. Room air is taken into each conditioned airunit 150 near an upper portion thereof. This room air comprises in partexhausted air from the “hot” air aisles of the data center defined byopposing air outlet sides 130 of the electronics racks 110.

Due to the ever increasing airflow requirements through electronicsracks, and limits of air distribution within the typical computer roominstallation, recirculation problems within the room may occur. This isshown in FIG. 2 for a raised floor layout, wherein hot air recirculation200 occurs from the air outlet sides 130 of the electronics racks backto the cold air aisle defined by the opposing air inlet sides 120 of theelectronics rack. This recirculation can occur because the conditionedair supplied through tiles 160 is typically only a fraction of theairflow rate forced through the electronics racks by the air movingdevices disposed therein. This can be due, for example, to limitationson the tile sizes (or diffuser flow rates). The remaining fraction ofthe supply of inlet side air is often made up by ambient room airthrough recirculation 200. This recirculating flow is often very complexin nature, and can lead to significantly higher rack unit inlettemperatures than might be expected.

The recirculation of hot exhaust air from the hot aisle of the computerroom installation to the cold aisle can be detrimental to theperformance and reliability of the computer system(s) or electronicsystem(s) within the racks. Data center equipment is typically designedto operate with rack air inlet temperatures in the 18-35° C. range. Fora raised floor layout such as depicted in FIG. 1, however, temperaturescan range from 15-20° C. at the lower portion of the rack, close to thecooled air input floor vents, to as much as 45-50° C. at the upperportion of the electronics rack, where the hot air can form aself-sustaining recirculation loop. Since the allowable rack heat loadis limited by the rack inlet air temperature at the “hot” part, thistemperature distribution correlates to an inefficient utilization ofavailable air conditioning capability. Computer installation equipmentalmost always represents a high capital investment to the customer.Thus, it is of significant importance, from a product reliability andperformance view point, and from a customer satisfaction and businessperspective, to achieve a substantially uniform temperature across theair inlet side of the rack unit. The efficient cooling of such computerand electronic systems, and the amelioration of localized hot air inlettemperatures to one or more rack units due to recirculation of aircurrents, are addressed by the apparatuses and methods disclosed herein.

FIG. 3 depicts one embodiment of a cooled electronics system, generallydenoted 300, in accordance with an aspect of the present invention. Inthis embodiment, electronics system 300 includes an electronics rack 310having an inlet door 320 and an outlet door 330, which respectively haveopenings to allow for the ingress and egress of external air,respectively, through the air inlet side and air outlet side ofelectronics rack 310. The system further includes at least oneair-moving device 312 for moving external air across at least oneelectronics subsystem 314 positioned within the electronics rack.Disposed within outlet door 330 is an air-to-liquid heat exchanger 340across which the inlet-to-outlet airflow through the electronics rackpasses. A cooling unit 350 is used to buffer the air-to-liquid heatexchanger from facility coolant 360, for example, provided via acomputer room water-conditioning unit (not shown). Air-to-liquid heatexchanger 340 removes heat from the exhausted inlet-to-outlet airflowthrough the electronics rack via the system coolant, for ultimatetransfer in cooling unit 350 to facility coolant 360 vialiquid-to-liquid heat exchanger 352 disposed therein. This coolingapparatus advantageously reduces heat load on existing air-conditioningunits within the data center, and facilitates cooling of electronicsracks by cooling the air egressing from the electronics rack and thuscooling any air recirculating to the air inlet side thereof.

As shown in FIG. 3, a system coolant loop 345 couples air-to-liquid heatexchanger 340 to cooling unit 350. In one embodiment, the system coolantemployed is water. By way of example, such a system is described in U.S.Pat. No. 7,385,810 B2, issued Jun. 10, 2008, and entitled “Apparatus andMethod for Facilitating Cooling of an Electronics Rack Employing a HeatExchange Assembly Mounted to an Outlet Door Cover of the ElectronicsRack”.

In this co-pending application, the inlet and outlet plenums mountwithin the door and are coupled to supply and return manifolds disposedbeneath a raised floor. Presented hereinbelow are enhanced variations onsuch an outlet door heat exchanger. Specifically, disclosed hereinbelowis an air-to-liquid heat exchanger which employs a pumped refrigerant asthe system coolant. Connection hoses for the pumped refrigerant systemare, in one embodiment, metal braided hoses, and the system coolantsupply and return headers for the pumped refrigerant system are mountedoverhead relative to the electronics racks within the data center. Thus,for the pumped refrigerant system described below, system coolant entersand exits the respective system coolant inlet and outlet plenums at thetop of the door and rack. Further, because pumped refrigerant isemployed, the hose and couplings used in the pumped refrigerant systemsdescribed below are affixed at both ends, i.e., to the system coolantplenums on one end and to the overhead supply and return headers on theother end.

FIG. 4 is a plan view of one embodiment of a data center, generallydenoted 400, employing cooled electronics systems, in accordance with anaspect of the present invention. Data center 400 includes a plurality ofrows of electronics racks 310, each of which includes an inlet door 320and a hinged outlet door 330, such as described above in connection withthe embodiment of FIG. 3. Each outlet door 330 supports an air-to-liquidheat exchanger and system coolant inlet and outlet plenums as describedfurther hereinbelow. Multiple cooling units 350, referred to hereinbelowas pumping units, are disposed within the data center (along with one ormore air-conditioning units (not shown)). In this embodiment, eachpumping unit forms a system coolant distribution subsystem with one rowof a plurality of electronics racks. Each pumping unit includes aliquid-to-liquid heat exchanger where heat is transferred from a systemcoolant loop to a facility coolant loop. Chilled facility coolant, suchas water, is received via facility coolant supply line 401, and isreturned via facility coolant return line 402. System coolant, such asrefrigerant, is provided via a system coolant supply header 410extending over the respective row of electronics racks, and is returnvia a system coolant return header 420 also extending over therespective row of electronics racks. In one embodiment, the systemcoolant supply and return headers 410, 420 are hard-plumbed within thedata center, and preconfigured to align over and include branch linesextending towards electronics racks of a respective row of electronicsracks.

FIG. 5 depicts one embodiment of a cooling unit 350 for the data center400 of FIG. 4. Liquid-to-liquid heat exchanger 352 condenses avapor-liquid refrigerant mixture passing through the system coolant loopcomprising system coolant supply header 410 and system coolant returnheader 420. (In one embodiment, the system coolant has undergone heatingand partial vaporization within the respective air-to-liquid heatexchangers disposed within the outlet doors of the electronics racks.)The facility coolant loop of liquid-to-liquid heat exchanger 352comprises facility coolant supply line 401 and facility coolant returnline 402, which in one embodiment, provide chilled facility water to theliquid-to-liquid heat exchanger. A control valve 501 may be employed infacility coolant supply line 401 to control facility coolant flow ratethrough the liquid-to-liquid heat exchanger 352. After the vapor-liquidrefrigerant mixture condenses within liquid-to-liquid heat exchanger352, the condensed refrigerant is collected in a condensate reservoir510 for pumping via a redundant pump assembly 520 back to the respectiverow of electronics racks via system coolant supply header 410. As shownin FIG. 5, a bypass line 530 with a bypass valve 531 may be employed tocontrol the amount of system coolant fed back through the system coolantsupply header, and hence, control temperature of system coolantdelivered to the respective air-to-liquid heat exchangers mounted to thedoors of the electronics racks.

FIGS. 6 & 7 depict one embodiment of outlet door 330 supportingair-to-liquid heat exchanger 340, and system coolant inlet and outletplenums 601, 701. Referring to both figures collectively, outlet doorframe 331 supports a rigid flap 600, which attaches, for example, bybrazing or soldering, to a plate 710 secured between the system coolantinlet plenum 601 and system coolant outlet plenum 701.

In FIG. 6, right angle bend 610 is shown disposed at the top of systemcoolant inlet plenum 601. This right angle bend defines a horizontalinlet plenum portion, which extends above the top of door 330. Thecoolant inlet to system coolant inlet plenum 601 is coupled to a connectcoupling 611 for facilitating connection thereof to the respectivesupply hose, as described above. The air-to-liquid heat exchangercomprises a plurality of horizontally-oriented heat exchange tubesections 620. These heat exchange tube sections 620 each comprise acoolant channel having an inlet and an outlet, with each coolant channelbeing coupled to the system coolant inlet plenum 601 and each coolantchannel outlet being coupled to the system coolant outlet plenum 701. Aplurality of fins 630 are attached to horizontally-oriented heatexchange tube sections 620 for facilitating transfer of heat from airpassing across the air-to-liquid heat exchanger to coolant flowingthrough the plurality of heat exchange tube sections 620. In oneembodiment, the plurality of fins are vertically-oriented, rectangularfins attached to horizontally-oriented heat exchange tube sections 620.

Due to the low saturation (boiling) temperature of liquid refrigerant,removal of a heat load exiting the back of an electronics rack via therefrigerant will cause the refrigerant to vaporize within the heatexchange tube sections of the air-to-liquid heat exchanger, resulting intwo-phase flow and latent heat transfer. Two-phase latent heat transferis very effective as a heat removal method; however, problems occur inthe area of refrigerant flow distribution within the air-to-liquid heatexchanger and across multiple air-to-liquid heat exchangers of the datacenter due to vaporization of the refrigerant.

For example, within an air-to-liquid heat exchanger at the air outletside of an electronics rack such as described above, liquid refrigerantis pumped into a vertical supply plenum, from which the refrigerantflows through several parallel heat exchange tube sections spanning thewidth of the air-to-liquid heat exchanger, eventually mixing in thevertical return plenum. As a result of slightly lower refrigerant flowrates in the lower heat exchange tube sections of the air-to-liquid heatexchanger caused by pressure drops due to pipe fittings and friction,refrigerant flowing through these lower sections will have a tendency tovaporize first upon introduction of a (uniform) heat load to theair-to-liquid heat exchanger.

When liquid refrigerant vaporizes in one of the heat exchange tubesections due to an applied heat load, the pressure drop experiencedacross that heat exchange tube section will equal several times themagnitude of the pressure drop experienced by single-phase liquidrefrigerant flowing through the tube section. This increased pressuredrop creates a “resistance” for the refrigerant to flow in the lowertube sections where two-phase latent heat transfer is occurring. Asliquid flows through the coolant inlet plenum, with several parallelpaths to choose from, more liquid will flow through the tube sectionwith the least resistance, that is, the lowest pressure drop. It hasbeen observed through testing that latent heat removal affects increasefrom the upper sections of the rear door heat exchanger to the lowersections thereof. The greater the degree of vaporization due toincreased latent heat transfer occurring in the lower heat exchange tubesections, the larger the pressure drop, which causes a mal-distributionof refrigerant flow through the heat exchanger (and higher coolantpumping power consumption). Increased amounts of liquid bypass the lowersections of the rear door heat exchanger, where latent heat transfer isoccurring, resulting in increased single-phase liquid flow through theupper heat exchange tube sections and decreased two-phase flow throughthe lower heat exchange tube sections of the air-to-liquid heatexchanger. Single-phase refrigerant flow does not provide the desiredheat removal effects of latent heat transfer, and thus is to be avoided.

It is one goal of the present invention to develop an effectivemechanism for eliminating mal-distribution of refrigerant flow throughmultiple electronics racks of a data center, as well as within a heatexchange assembly between the heat exchange tube sections thereof toenhance heat transfer and/or minimize coolant pumping requirements.

FIGS. 8-10 address coolant flow mal-distribution between electronicsracks of a data center, while FIGS. 11-18 address mal-distribution ofcoolant flow between heat exchange tube sections of a heat exchangeassembly coupled to or associated with an electronics rack.

Referring first to FIG. 8, a data center 800 is illustrated comprising aplurality of electronics racks 810 and a coolant distribution unit 820.Each electronics rack 810 includes a heat exchange assembly 811, such asdescribed herein. Specifically, heat exchange assembly 811 includes anair-to-liquid heat exchanger, a coolant inlet plenum, and a coolantoutlet plenum, with multiple heat exchange tube sections of theair-to-liquid heat exchanger being coupled in parallel between thecoolant inlet plenum and the coolant outlet plenum. The coolantdistribution unit 820 may comprise, for example, a cooling unit such asdescribed above in connection with FIG. 5. Within coolant distributionunit 820, a liquid-to-liquid heat exchanger is employed to facilitatetransfer of heat from system coolant to facility coolant passing throughthe coolant distribution unit (via facility coolant supply line 801 andfacility coolant return line 802). As illustrated in FIG. 8, a mainsystem coolant supply line 805 supplies cooled system coolant to acoolant supply manifold 821, and a main coolant return line 806 receivesexhausted system coolant via a coolant return manifold 822. A pluralityof coolant supply lines 823 and a plurality of coolant return lines 824facilitate coupling coolant distribution unit 820 in fluid communicationwith the plurality of heat exchange assemblies 811 associated with theelectronics racks 810. In one embodiment, quick connect couplings 812are employed to connect the individual coolant supply lines and coolantreturn lines to the respective heat exchange assemblies 811. By way ofexample, these quick connect couplings may comprise various types ofcommercially available couplings, such as those available from ColderProducts Company, of St. Paul, Minn., USA, or Parker Hannifin, ofCleveland, Ohio, USA.

FIG. 9A illustrates potential mal-distribution of system coolant flowthrough the heat exchange assemblies of the electronics racks using theclosed system coolant loop configuration of FIG. 8. As illustrated, dueto the effects of two-phase flow on pressure drop, for an overallconstant pressure drop across the heat exchange assemblies, anelectronics rack at no information technology (IT) load experiences thehighest coolant flow rate through its associated heat exchange assembly,while an electronics rack at full IT load experiences the lowest coolantflow rate through its heat exchange assembly. These flow characteristicsfor the different heat load conditions cause undesirable refrigerantflow mal-distribution at the data center level.

Returning to FIG. 8, a plurality of flow restrictors 825 are provided inassociation with the plurality of coolant supply lines 823 (in oneembodiment), with each flow restrictor being associated with arespective coolant supply line 823 for tailoring coolant flow resistancethrough that line. In addition, pressure and temperature sensors 826 areprovided (in one embodiment) on each coolant supply line 823 and eachcoolant return line 824. Flow restrictors 825 are, by way of example,adjustable flow restrictors, each of which comprises a dynamicallyadjustable orifice opening size for tailoring coolant flow resistancethrough the respective coolant line, and thus through the respectiveheat exchange assembly of the associated electronics rack based on itsheat load.

By dynamically adjusting the orifice opening sizes of the adjustableflow restrictors, a cooling apparatus is provided which is able totailor (or adjust) coolant flow through the respective heat exchangeassemblies, for example, based on the current IT loads of the associatedelectronics racks. This is illustrated in FIG. 9B. As shown, the flowrestrictors are controlled so that, for a constant overall pressure dropacross the heat exchange assemblies associated with the plurality ofelectronics racks in the data center, an electronics rack at no IT loadreceives the lowest (or no) coolant flow rate, while an electronics rackat full IT load receives the highest coolant flow rate through itsrespective air-to-liquid heat exchanger. This adjusting of the flowresistance is significant in a system which employs refrigerant andlatent heat transfer, such as proposed herein.

FIG. 10 illustrates one embodiment of logic for controlling theadjustable flow restrictors illustrated in FIG. 8, as well as overallcoolant flow. Initially, each adjustable flow restrictor (i.e., eachrack-level adjustable flow restrictor) is set to 50% open position 1000,and the rack-level IT loads are obtained from the electronics racks ormeasured using power measurement devices 1010. This rack-level load (orpower consumed) information is provided to a data center control unit,for example, disposed within the one or more coolant distribution units(CDUs) of the data center 1020. Based on the total rack powerutilizations, the control unit estimates the total refrigerant flowrequired and sets the pump speed of the CDUs to force the estimatedcoolant flow through the data center coolant distribution loop 1030. Therack-level adjustable flow restrictor for the highest power-consumingelectronics rack is set to full open position 1040, and coolant pressureand temperature information is collected from the coolant supply andreturn measurement locations 1050, for example, from pressure andtemperature sensors associated with the plurality of coolant supplylines and the plurality of coolant return lines, such as discussed abovein connection with FIG. 8. The rack-level adjustable flow restrictorsfor the remaining heat exchangers are then set to ensure that theexhausting coolant from each heat exchange assembly is in asuper-heated, thermodynamic state within a specified range ofsuper-heated temperatures, based on the pressure and temperature datafor that rack 1060. Logic checks the flow through the data centerdistribution coolant loop at the coolant distribution unit(s) andadjusts the pump speed to force the estimated required flow through theloop based on the total of the rack heat loads 1070, and returns (forexample, after a defined time delay) to collect the current rack-levelpower utilizations from the electronics racks 1010, before repeating theprocess.

Note that pressure and temperature sensors 826 are provided in theplurality of coolant supply lines 823 and the plurality of coolantreturn lines 824 in the data center embodiment illustrated in FIG. 8.These pressure and temperature sensors allow for the determination ofthe thermodynamic state of the refrigerant as it enters and exits theheat exchange assemblies 811 and associated flow restrictors 825. It isdesirable for the coolant exiting the heat exchanger subassembly 811 tobe slightly super-heated, that is, with no liquid content. Pressure(P)-enthalpy (H) diagrams for R-134a refrigerant are available in theliterature, which indicate the regions in which such a refrigerant issub-cooled, saturated, or super-heated. These diagrams (or functions)utilize variables, such as pressure and temperature (or enthalpy if thequality of the two-phase mixture needs to be known). Thus, thethermodynamic state of the coolant can be determined and controlledusing pressure and temperature data. The pressure and temperature valuesmeasured will be input into a coolant-dependant algorithm (defined bythe P-H diagram/properties of the coolant) that determines if thecoolant is super-heated. This algorithm can be readily ascertained byone skilled in the art.

If the coolant is not super-heated (i.e., the coolant is sub-cooled orin a two-phase saturated condition), the algorithm will modulate theadjustable flow restrictors 825 associated with the heat exchangeassemblies until the exiting coolant is super-heated. This ensures thatall coolant exiting the heat exchanger has utilized its latent coolanteffects and there is a 100% vapor in the return plenum. The modulationof the adjustable orifices serves to increase the flow resistance, andthus, redirects coolant flow to ensure sufficient vaporization andcooling in all sections of the heat exchange assemblies. If the heatload of a specific electronics rack that has a low coolant flow suddenlyincreases, then the extent of super-heat will be determined using thesame pressure and temperature sensor information. If the degree ofsuper-heat is too much, then the controller will open the respectiveflow restrictor, thereby reducing the flow resistance through the heatexchange assembly and thus attracting more coolant flow, therebyreducing the degree of super-heat. Thus, one skilled in the art willnote that the control algorithm employed can determine the thermodynamicstate using pressure and temperature data, manipulate the flowrestrictor to force a super-heated condition, and also force the degreeof super-heat on as to be within a specific temperature differential inexcess of the saturated condition. For example, if for a specificdesign, the saturated temperature of the refrigerant flow is 18° C.,then the flow restrictor may be controlled to force the exhaustrefrigerant vapor to be at 20° C.

Various actively controlled, adjustable flow restrictors are availablein the art. For example, reference the EX4 or EX6 refrigerant flowcontrol valves offered by Emerson Electric Company, of St. Louis, Mo.,U.S.A.

FIGS. 11-18 depict a further aspect of the present invention, whereinone or more flow restrictors are employed within a rear door,air-to-liquid heat exchanger (such as described above) for tailoringcoolant flow resistance through one or more heat exchange tube sectionsof the air-to-liquid heat exchanger to enhance overall heat transferacross the multiple heat exchange tube sections. In one example, the oneor more flow restrictors ensure that vaporization occurs within eachtube section of the multiple tube sections of the heat exchanger for agiven operating condition or range of conditions. By achieving this,flow resistance gradients that might otherwise exist within the reardoor heat exchanger are eliminated, allowing for a more uniformrefrigerant flow and consistent latent heat transfer in the tubesections. Once latent heat removal occurs roughly equally within theheat exchange tube sections (for uniform heat loads) of the rear doorheat exchanger, greater heat removal is realized.

Various installations of flow restrictors within a rear door heatexchanger are described below. In a system where the rear door heatexchanger (or multiple rear door heat exchangers) receives refrigerantpumped from a coolant distribution unit, the refrigerant should bemaintained as a sub-cooled liquid through the supply lines incommunication with the rear door heat exchanger(s). Once the sub-cooledliquid (refrigerant) reaches its saturation pressure for a giventemperature, the liquid begins to vaporize. To bring sub-cooledrefrigerant into saturation, a flow restrictor (such as described abovein connection with FIG. 8) may be employed within the supply and/orreturn lines immediately before and/or after the rear door heatexchanger, which is designed to create a pressure drop to bring therefrigerant to saturation before entering the heat exchanger. Thismethod ensures that the refrigerant is on the verge of vaporization asdelivered to the coolant inlet plenum of the heat exchanger.

To further facilitate heat transfer across the heat exchange tubesections of the rear door heat exchanger, at least one fixed (oradjustable) flow restrictor is provided for each tube section, asillustrated in FIGS. 11 & 12. In one embodiment, these fixed oradjustable flow restrictors are disposed at the coolant channel inlets(and/or coolant channel outlets) to the respective heat exchange tubesections of the heat exchanger.

Referring collectively to FIGS. 11 & 12, and as noted above, rack outletdoor 330 supports air-to-liquid heat exchanger 340, and system coolantinlet and outlet plenums 601, 701. Outlet door frame 331 supports arigid flap 600, which attaches, for example, by brazing or soldering, toa plate 710 secured between the system coolant inlet plenum 601 andsystem coolant outlet plenum 701. In FIG. 11, a right angle bend 610 isshown disposed at the top of system coolant inlet plenum 601. This rightangle bend defines a horizontal inlet plenum portion which extends abovethe top of door 330, which facilitates attaching a supply hose to thehinged outlet door. The air-to-liquid heat exchanger comprises aplurality of horizontally-oriented heat exchange tube sections 620.These heat exchange tube sections 620 each comprise a coolant channelhaving an inlet and an outlet, with each coolant channel inlet beingcoupled to the system coolant inlet plenum 601 and each coolant channeloutlet being coupled to the system coolant outlet plenum 701. Aplurality of fins 630 are attached to the horizontally-oriented heatexchange tube sections 620 for facilitating transfer of heat from airpassing across the air-to-liquid heat exchanger to coolant flowingthrough the plurality of heat exchange tube sections 620, in oneembodiment, the plurality of fins are vertically-oriented, rectangularfins attached to horizontally-oriented heat exchange tube sections 620.

As illustrated in FIGS. 11 & 12, a plurality of flow restrictors 1100are provided at the heat exchange tube sections. In this embodiment,each heat exchange tube section 620 has a first flow restrictor at itscoolant channel inlet and a second flow restrictor at its coolantchannel outlet (by way of example only). These flow restrictors maycomprise any desired combination of fixed or adjustable flow restrictorsto accomplish the desired tailoring of coolant flow resistance throughthe respective heat exchange tube sections. In this initial example, theflow restrictors are assumed to comprise fixed orifice diameters, withat least two fixed orifice diameters of the flow restrictors beingdifferently sized to define different coolant flow resistances throughat least two heat exchange tube sections of the multiple heat exchangetube sections of the rear door heat exchanger. By defining differentcoolant flow resistances, the multiple flow restrictors tailor coolantflow to facilitate overall heat transfer within the multiple heatexchange tube sections of the air-to-liquid heat exchanger, for example,by facilitating vaporization of refrigerant within each of the heatexchange tube sections, or by equalizing flow across the heat exchangetube sections of the rear door heat exchanger, notwithstanding thepresence of heat transfer gradients across the heat exchange tubesections. In an alternative embodiment, the flow restrictors may againcomprise fixed orifice diameters (or opening sizes), with each orificeopening size of the flow restrictors being identical to ensure a commoncoolant flow through the multiple heat exchange tube sections of therear door heat exchanger. This implementation might be advantageouswhere there is uniform heat flux across the heat exchange tube sections.

FIG. 13 depicts an alternate embodiment of an apparatus for facilitatingcooling or removal of heat from an electronics rack, in accordance withan aspect of the present invention. This cooling apparatus is (in oneembodiment) a rear door heat exchanger (such as described above) whereinonly a portion of the heat exchanger is illustrated, including systemcoolant inlet plenum 601 and heat exchange tube sections 620. In thisembodiment, multiple flow restrictors 1300 are illustrated at thecoolant channel inlets of the respective heat exchange tube sections620. Also, these flow restrictors 1300 are shown to comprise differentorifice diameters (or opening sizes), wherein the apparatus transitionsfrom a smaller orifice diameter 1310 a to a larger orifice diameter 1310f, progressing downwards from the top of the rear door heat exchangertowards the bottom. Using different orifice diameters within the flowrestrictors produces different flow resistances, and for example,different magnitudes of flow resistances across the heat exchange tubesections. These different orifice diameters may be tailored to ensureequivalent or desired amounts of refrigerant flow to facilitatevaporization within each heat exchange tube section. Varying amounts ofpower consumption (or heat load) may be applied to the rear door heatexchanger from the associated electronics rack, and thus, varying theorifice diameters based on location of the heat exchange tube sectionsfacilitates accommodating a range of heat load configurations.

FIG. 14 depicts a partial rear door heat exchange apparatus forfacilitating cooling of exhaust air at the air outlet side of anelectronics rack. The apparatus includes an air-to-liquid heat exchangercomprising a coolant inlet plenum 601, a coolant outlet plenum 701 andmultiple heat exchange tube sections 620′, only one of which isillustrated in FIG. 14. Each heat exchange tube section includes asinusoidal coolant channel formed, in this example, from a plurality ofstraight channels with U-shaped bends attached to the ends thereof, andincluding a coolant channel inlet 1401 and coolant channel outlet 1411.Each coolant channel inlet 1401 is in fluid communication with coolantinlet plenum 601, and each coolant channel outlet 1411 is in fluidcommunication with coolant outlet plenum 701. As illustrated, multiplebraze points 1420 are employed during one manufacturing embodiment ofthe sinusoidal heat exchange tube section to attach the straight channelportions to the U-shaped portions, as well as to attach the coolantchannel inlet and coolant channel outlet to the respective plenums.

During fabrication of the rear door heat exchanger, a first flowrestrictor 1400 can be placed into the heat exchange tube section at thecoolant channel inlet, and a second flow restrictor 1410 can be placedin the heat exchange tube section at the coolant channel outlet. Asexplained further below, these flow restrictors may be brazed or crimpedinto position, followed by the normal brazing 1420 of the straightchannel sections used in forming the desired heat exchange tube sectionconfiguration. By way of example only, first flow restrictor 1400 andsecond flow restrictor 1410 have fixed diameter orifices selected toadjust the flow resistance through the respective heat exchange tubesection based on testing of the heat exchange design with two-phaserefrigerant heat transfer. Note that although illustrated in FIG. 14 asincluding two flow restrictors, that is, one at the coolant channelinlet and one at the coolant channel outlet, the cooling apparatusdisclosed herein could employ one flow restrictor of fixed orificediameter (or adjustable orifice opening size) at either the coolantchannel inlet or coolant channel outlet of the heat exchange tubesection, or more than two flow restrictors disposed throughout the heatexchange tube section.

FIGS. 15A & 15B depict alternate embodiments of a fixed orifice diameterflow restrictor, which may be used (for example) at the coolant channelinlet or coolant channel outlet of one or more heat exchange tubesections of the rear door heat exchanger described above.

In FIG. 15A, a flow restrictor disk 1500 is shown positioned in a heatexchanger tube section 620′, for example, by brazing or other means ofattaching the perimeter of the disk to the inner wall of the heatexchange tube section (which is assumed to be fabricated of metal). Flowrestrictor 1500 has an orifice 1510 extending therethrough with a fixedorifice diameter D.

In the alternate embodiment of FIG. 15B, a cylindrical-shaped flowrestrictor 1550 is illustrated crimped 1555 in position within a heatexchange tube section 620′, for example, at the coolant channel inlet orcoolant channel outlet thereof. This cylindrical flow restrictorincludes a cylindrical-shaped orifice 1560 extending therethrough. Inthe embodiment illustrated, the cylindrical-shaped orifice 1560 has aconstant, fixed diameter D.

By way of specific example, a cylindrical flow restrictor such asdepicted in FIG. 15B may be bored out to form a shell that may beinserted directly into an existing heat exchange tube section during themanufacturing process thereof. In one example, the heat exchange tubesection utilizes ⅜ inch piping, and the cylindrical flow restrictor maybe held in position via crimping to constrict the piping around thecylindrical flow restrictor. As with the disk-type flow restrictor,insertion of the cylindrical flow restrictor into the rear door heatexchanger piping is a simple manufacturing operation, is inexpensive andis a cost effective approach to achieving the desired tailoring of flowresistance through the respective heat exchange tube section. Thefabrication method described herein ensures that all coolant flowsthrough the orifice, resulting in the target flow restriction.Cylindrical flow restrictors containing different orifice diameters maybe employed throughout the rear door heat exchanger to achieve thedesired tailoring of coolant flow resistances through the heat exchangetube sections to enhance overall heat transfer within the multiple heatexchange tube sections of the air-to-liquid heat exchanger.

To satisfy changing cooling requirements across a rear door heatexchanger or between multiple rear door heat exchangers (as discussedabove in connection with FIGS. 8-10), adjustable flow restrictors may beemployed.

FIG. 16 illustrates one embodiment of a passively controlled, adjustableflow restrictor 1600 at the coolant channel inlet 1401 of heat exchangetube section 620′ coupling the tube section to coolant inlet plenum 601.As illustrated, heat exchange tube section 620′ also couples to coolantoutlet plenum 701 via its coolant channel outlet 1411. A plurality offins 630 are attached to the horizontally-oriented heat exchange tubesections 620′ for facilitating transfer of heat from air passing acrossthe air-to-liquid heat exchanger to coolant flowing through theplurality of heat exchange tube sections 620′ (only one of which isillustrated in the figure). In this embodiment, passively controlledadjustable flow restrictor 1600 comprises a thermal sensor 1610 forsensing temperature or exhaust airflow passing across the heat exchangetube section 620′. The sensed temperature 1610 is fed back 1601 to thepassively controlled adjustable flow restrictor 1600.

As a specific example, temperature sensor 1610 might comprise a thermalsensing bulb and pneumatic/spring-actuated wire inserted into the airstream and coupling back to the passively controlled, adjustable flowrestrictor 1600 located, for example, adjacent to the coolant channelinlet of true heat exchange tube section 620′. As a specific example,the passively controlled, adjustable flow restrictor might comprise athermostatic-actuated valve, such as provided by Metrix Valve Corp. ofGlendora, Calif., USA. This configuration provides the advantage thateach heat exchange tube section is self-monitoring and adjusts thecoolant flow resistance therethrough as required to cool the heat loadpassing across that tube section. No additional power or wiring isrequired to achieve the automated control. Additionally, the passivelycontrolled, adjustable flow restrictor is reverse-acting in that astemperature of airflow across the tube section drops, the flowrestrictor automatically at least partially closes, producing a greaterpressure drop across, and lower coolant flow through, the heat exchangetube section.

FIG. 17 depicts an alternate embodiment of a cooling apparatus inaccordance with an aspect of the present invention. In this embodiment,the apparatus again comprises an air-to-liquid heat exchanger having acoolant inlet plenum 601, coolant outlet plenum 701 and multiple heatexchange tube sections 620′ (only one of which is shown). A plurality offins 630 are attached to the horizontally-oriented heat exchange tubesections 620′ for facilitating transfer of heat from air passing acrossthe air-to-liquid heat exchanger to coolant flowing through the heatexchange tube sections 620′. In the embodiment illustrated, an activelycontrolled, adjustable flow restrictor 1700 is illustrated, such asdescribed above in connection with the cooling apparatus of FIGS. 8-10.In this embodiment, the actively controlled, adjustable flow restrictor1700 is disposed at the coolant channel inlet 1401 of heat exchange tubesection 620′, and includes an adjustable orifice of varying opening sizewhich is actively controlled (e.g., by a rack-level control unit (notshown)). In addition, pressure and temperature sensors 1710 and 1720 areprovided at the coolant channel inlet 1401 and coolant channel outlet1411, respectively, of heat exchange tube section 620′. The rack-levelcontrol unit employs the sensed pressure and temperature readings forsystem coolant passing through the multiple heat exchange tube sectionsin determining whether to increase or decrease orifice opening sizeswithin the actively controlled, adjustable flow restrictors associatedwith the multiple heat exchange tube sections, and thereby tailorcoolant flow resistance through the heat exchange tube sections. Thistailoring of coolant flow is controlled to enhance heat transfer withinthe multiple heat exchange tube sections, for example, by ensuring thatlatent heat transfer to refrigerant occurs within each heat exchangetube section.

FIG. 18 illustrates one embodiment of logic for controlling adjustableflow restrictors in a heat exchange assembly, such as depicted in FIG.17. Initially, each adjustable flow restrictor (i.e., each tube sectionlevel flow restrictor within the heat exchanger) is set to 50% openposition 1800, and the power consumption of the rack unit subsystems inopposing relation to the individual heat exchange tube sections iscollected from the corresponding subsystems or is ascertained usingpower measurement devices 1810. This sectional rack power information isthen employed by a rack-level control unit 1820 (for example). The logicsets the adjustable flow restrictor associated with the heat exchangetube section that is in opposing relation to the highest power consumingsubsystem in the electronics rack to a full open position 1830. This isthe heat exchange tube section which will experience the highest exhaustairflow temperature from the electronics rack. Next, the logic sets theadjustable flow restrictors for the remaining heat exchange tubesections to ensure that the exhausting system coolant from each heatexchange tube section is in a super-heated thermodynamic state within aspecific range of super-heat temperatures, based on the sensed pressureand temperature data for the individual heat exchange tube sections1840, and then returns to collect updated rack subsection powerconsumption information 1810.

Note that in the embodiment of FIG. 17, pressure and temperature sensor1720 is disposed at the coolant channel outlet 1411 of heat exchangetube section 620 to measure the temperature and pressure of the coolantexiting the heat exchange tube section. From this data, thethermodynamic state of the coolant can be determined within eachsection. It is also desired that the coolant exiting each of theparallel-coupled heat exchange tube sections be slightly super-heated(i.e., above saturation temperature with no liquid content). Measuringtemperature and pressure at the outlet of the heat exchange tubesections provides a mechanism for determining if the coolant exiting theheat exchange tube sections is super-heated. Pressure-enthalpy (P-H)diagrams for R-134a refrigerant are available in the art, which indicatethe regions in which the coolant is saturated, as well as super-heated,by the variables of pressure, temperature and enthalpy. Pressure andtemperature measurement of the coolant provides sufficient data todetermine if the coolant is saturated or super-heated. The pressure andtemperature values measured will be input to a coolant-dependantalgorithm (defined by the P-H diagram/properties of the coolant), whichdetermines if the coolant is super-heated. If the coolant is notsuper-heated (i.e., coolant is sub-cooled or saturated), then theadjustable orifice opening size will be modulated until the exitingcoolant is super-heated. This ensures that all coolant exiting each heatexchange tube section has utilized its latent cooling effect and is 100%vapor in the return plenum. The modulation of the adjustable orificeopening size serves to increase the flow resistance in a single heatexchange tube section, while redirecting coolant flow to ensuresufficient vaporization and cooling in all sections of the rear doorheat exchanger. If the heat load across a specific heat exchange tubesection which has a low coolant flow suddenly increases, then the extentof super-heat will be determined using the same pressure and temperaturesensor information. If the degree of super-heat is too much, then thecontrol algorithm opens the valve, thereby reducing the flow resistanceof the heat exchange tube section, and thus attracting more coolant flowand reducing the degree of super-heat. Therefore, the controllerdetermines the thermodynamic state using the pressure and temperaturedata, manipulates the valve position to force a super-heated condition,and also forces the degree of super-heat so as to be within a specifictemperature differential in excess of the saturated condition. Forexample, if for a specific design, the saturated temperature of therefrigerant flow is 118° C., then the valve may be controlled to forcethe exhaust refrigerant vapor to be at 20° C.

Further details and variations of liquid-based cooling apparatuses andmethods for cooling electronics systems and/or electronics racks aredisclosed in co-filed U.S. patent application Ser. No. 12/556,019,entitled “Pressure Control Unit and Method Facilitating Single-PhaseHeat Transfer in a Cooling System”, published Mar. 10, 2011, as U.S.Patent Publication No. 2011/0058637 A1 co-filed U.S. patent applicationSer. No. 12/556,031, entitled “Control of System Coolant to FacilitateTwo-Phase Heat Transfer in a Multi-Evaporator Cooling System”, publishedMar. 10, 2011, as U.S. Patent Publication No. 2011/0056225 A1; co-filedU.S. patent application Ser. No. 12/056,053, entitled “System and Methodfor Facilitating Parallel Cooling of Liquid-Cooled Electronics Racks”,published Mar. 10, 2011, as U.S. Patent Publication No. 2011/0056674 A1;and co-tiled U.S. patent application Ser. No. 12/556,066 entitled“Cooling System and Method Minimizing Power Consumption in CoolingLiquid-Cooled Electronics Racks”, published Mar. 10, 2011, as U.S.Patent Publication No. 2011/0060470 A1, the entirety of each of which ishereby incorporated herein by reference.

As will be appreciated by one skilled in the art, aspects of thecontroller described above may be embodied as a system, method orcomputer program product. Accordingly, aspects of the controller maytake the form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.) oran embodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit”, “module” or “system”.Furthermore, aspects of the controller may take the form of a computerprogram product embodied in one or more computer readable medium(s)having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, or semiconductorsystem, apparatus, or device, or any suitable combination of theforegoing. More specific examples (a non-exhaustive list) of thecomputer readable storage medium include the following: an electricalconnection having one or more wires, a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), anoptical fiber, a portable compact disc read-only memory (CD-ROM), anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing. In the context of this document, acomputer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signalwith computer-readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer-readable signal medium may be any computer-readable medium thatis not a computer-readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus or device.

Program code embodied on a computer readable medium may be transmittedusing an appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programminglanguage, such as Java, Smalltalk, C++ or the like, and conventionalprocedural programming languages, such as the “C” programming languageor similar programming languages.

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowcharts or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblocks may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Although embodiments have been depicted and described in detail herein,it will be apparent to those skilled in the relevant art that variousmodifications, additions, substitutions and the like can be made withoutdeparting from the spirit of the invention and these are thereforeconsidered to be within the scope of the invention as defined in thefollowing claims.

What is claimed is:
 1. A cooling apparatus for a plurality ofelectronics racks, each electronics rack comprising a heat exchangeassembly, the cooling apparatus comprising: a coolant distribution unitfor supplying cooled system coolant to the heat exchange assemblies ofthe plurality of electronics racks; a plurality of coolant supply lines,each coolant supply line being in fluid communication with the coolantdistribution unit and the respective heat exchange assembly of arespective electronics rack of the plurality of electronics racks andfacilitating supply of system coolant from the coolant distribution unitto the respective heat exchange assembly; a plurality of coolant returnlines, each coolant return line coupling in fluid communication the heatexchange assembly of a respective electronics rack of the plurality ofelectronics racks with the coolant distribution unit and facilitatingreturn of exhausted system coolant from the respective heat exchangeassembly to the coolant distribution unit, wherein system coolantcirculates in a closed loop between the coolant distribution unit andthe heat exchange assemblies via, at least in part, the plurality ofcoolant supply lines and the plurality of coolant return lines; aplurality of inline flow restrictors associated with the plurality ofcoolant supply lines or the plurality of coolant return lines, eachinline flow restrictor having a dynamically adjustable orifice openingto control coolant flow resistance of the inline flow restrictor, andeach inline flow restrictor being associated with a respective coolantline of the plurality of coolant supply lines or the plurality ofcoolant return lines for tailoring coolant flow resistance through thatcoolant line, and thereby through the respective heat exchange assemblyof a respective electronics rack; and a controller programmed to controlsize of the dynamically adjustable orifice opening of each inline flowrestrictor of the plurality of inline flow restrictors to force thecoolant exiting the respective heat exchange assembly to be at aspecified super-heated temperature, above a saturation temperature ofthe coolant, the controller automatically: collecting rack powerutilization data for each electronics rack of the plurality ofelectronics racks; summing the rack power utilizations of the pluralityof electronics racks and determining a total coolant flow required tocool the plurality of electronics racks, and based on the total coolantflow required, setting a rate of cooled system coolant supplied by thecoolant distribution unit; determining the highest power utilizingelectronics rack of the plurality of electronics racks and setting theinline flow restrictor associated therewith to full open position;ascertaining coolant pressure and temperature information at the heatexchange assemblies of the plurality of electronics racks; and settinginline flow restrictors associated with the remaining electronics racksof the plurality of electronics racks to ensure coolant exhausting fromthe heat exchange assemblies of the remaining electronics racks is in asuper-heated vapor state within a specified range of super-heatedtemperatures based on the collected coolant pressure and temperatureinformation.
 2. The cooling apparatus of claim 1, wherein the pluralityof inline flow restrictors define different coolant flow resistancesthrough at least two heat exchange assemblies of the plurality ofelectronics racks.
 3. The cooling apparatus of claim 1, furthercomprising coolant pressure and temperature sensors associated with atleast one of the plurality of coolant supply lines or the plurality ofcoolant return lines for sensing pressure and temperature of systemcoolant passing therethrough, wherein sensed pressure and temperature ofthe system coolant facilitates dynamic adjustment of an adjustableorifice opening size of at least one adjustable inline flow restrictorby the controller to force the coolant exiting the respective heatexchange assembly to be at the specified, superheated temperature abovethe saturation temperature of the coolant.
 4. The cooling apparatus ofclaim 1, wherein the controller further adjusts pump speed of a coolantpump of the coolant distribution unit based on a total power consumptionof the plurality of electronics racks.
 5. The cooling apparatus of claim1, wherein at least one heat exchange assembly of the heat exchangeassemblies of the plurality of electronics racks comprises: anair-to-liquid heat exchanger comprising a coolant inlet plenum, acoolant outlet plenum, and multiple heat exchange tube sections coupledin parallel between the coolant inlet plenum and the coolant outletplenum, each heat exchange tube section comprising a coolant channelhaving a coolant channel inlet and a coolant channel outlet, eachcoolant channel inlet being in fluid communication with the coolantinlet plenum and each coolant channel outlet being in fluidcommunication with the coolant outlet plenum, and wherein system coolantflows through the multiple heat exchange tube sections in parallel; andat least one tube section flow restrictor associated with theair-to-liquid heat exchanger, each tube section flow restrictor of theat least one tube section flow restrictor being associated with arespective heat exchange tube section of the multiple heat exchange tubesections of the air-to-liquid heat exchanger and being disposed at oneof the coolant channel inlet or the coolant channel outlet thereof, theat least one tube section flow restrictor tailoring coolant flowresistance through the respective heat exchange tube section to enhanceheat transfer within the multiple heat exchange tube sections of theair-to-liquid heat exchanger.
 6. The cooling apparatus of claim 5,wherein the at least one air-to-liquid heat exchange assembly furthercomprises multiple tube section flow restrictors associated with themultiple heat exchange tube sections, each tube section flow restrictorof the multiple tube section flow restrictors being associated with arespective heat exchange tube section of the multiple heat exchange tubesections and being disposed at one of the coolant channel inlet or thecoolant channel outlet thereof, and wherein the multiple tube sectionflow restrictors tailor a coolant flow distribution differential throughthe multiple heat exchange tube sections, the coolant flow distributiondifferential being tailored based on at least one of an air temperaturedifferential across at least two heat exchange tube sections of themultiple heat exchange tube sections or location of the multiple heatexchange tube sections relative to the respective electronics rack. 7.The cooling apparatus of claim 5, wherein the at least one tube sectionflow restrictor defines different coolant flow resistances through atleast two heat exchange tube sections of the multiple heat exchange tubesections.
 8. The cooling apparatus of claim 5, wherein the at least onetube section flow restrictor comprises at least one adjustable tubesection flow restrictor, each adjustable tube section flow restrictor ofthe at least one adjustable tube section flow restrictor comprising adynamically adjustable orifice opening size for dynamically adjustingsystem coolant flow resistance through the tube section flow restrictor,and thereby through the respective heat exchange tube section of themultiple heat exchange tube sections.
 9. The cooling apparatus of claim1, wherein the controller further sets, prior to collecting the rackpower utilization data for each electronics rack, each inline flowrestrictor of the plurality of flow restrictors to 50% open position.